Add async connection testing via workers for security isolation - #62343

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Summary

Follows the direction proposed by @potiuk in #59643 to move connection testing off the API server and onto workers.

Connection testing has been disabled by default since Airflow 2.7.0 because executing user-supplied driver code (ODBC/JDBC) on the API server poses security risks, and workers typically have network access to external systems that API servers don't.

This moves the whole thing onto workers. A dedicated TestConnection workload goes through the scheduler, gets dispatched to a supporting executor, and the worker runs test_connection()` with a proper timeout. Results come back through the Execution API. Design was discussed on dev@ : "[DISCUSS] Move connection testing to workers" (Feb 2026).

Demo

Overview

• Dedicated workload type : not piggybacking on ExecuteCallback, so connection tests never compete with correctness-critical callbacks
• Scheduler dispatch + reaper: PENDING tests get dispatched to a supporting executor, capped by max_connection_test_concurrency (default 4). A reaper catches stuck tests after timeout + grace period
• Worker-side timeout : signal.alarm enforcement in LocalExecutor, results reported back via Execution API
• Request Buffer Mechanism: <TODO:add details>
• Queue parameter: optional queue field on the API, wired through to scheduler dispatch.
• Fail-fast: supports_connection_test flag on BaseExecutor, immediate FAILED if no executor supports it

Config

• [connection_test] timeout: worker timeout, default 60s
• [connection_test] max_concurrency: dispatch budget, default 4
• [connection_test] reaper_interval: reaper frequency, default 30s

Not in this PR

• UI changes (will create separate pr for this)

References

• Dev mailing list discussion: DISCUSS Move connection testing to workers

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.